This reverts commit 97ba98e.

This reverts commit 97ba98e.

This reverts commit 97ba98e.

…5117)

* feat: cleaning up old changes. making first changes to create the API pieces.

* feat: cleaning up old changes. making first changes to create the API pieces.

* feat: insert new filters.

* feat: insert new filters and order by params.

* feat: customizing provider map and serializer.

* fix: changing TIME_CHOICES options.

* feat: first unit tests.

* feat: wip.

* feat: fixing provider map.

* feat: fixing provider map.

* update ec2 annotations to get all required fields

* use AWSEC2ComputeQueryParamSerializer

* feat: creating orderby and groupby serializers for ec2

* feat: unit tests for filters

* feat: removing group_by - not needed on ec2

* feat: fixing orderby serializer and starting units tests

* fix: typo

* feat: changing usage_hours to usage_amount

* feat: fixing unit test.

* wip: blocking some filters and unit test.

* feat: unit tests for group by filter

* flake8 fix

* feat: inserting more filters on validate function.

* feat: updating validate method to use similar logic and add filters.

* fix: changing unit tests for some filters.

* feat: testing filter combinations and flake8 checks.

* fix: test

* feat: serializer Unit tests and view Unit test fix

* flake8 fix

* fix: new approach to satisfy CodeCov

* fix:: getting rid of validate custom method.

* fix: commenting tags.

* fix: validarte functions, tests.

* handle filter params for specific report type

* transform tags to desired ui format

* default to monthly resolution on the EC2 endpoint

* add special pagination for EC2

* use default report type time period settings if exists

* fix typo

* fix: fixing parameters validations.

* [COST-5141] Fix management command to use continue instead of return. (#5173)

* [COST-5128] Process new subs tagging strategy to identify non-converted instances (#5162)

* [COST-4745] Add data_transfer_direction to OCP on GCP Trino tables (#5130)

* [COST-4741] Add data_transfer_direction for AWS network costs to Trino tables (#5129)

* [COST-5168] - Adding new penalty pipeline (#5176)

* [COST-5168] - Adding new penalty pipeline

* Improve our logging readability (#5178)

* add prometheus metrics for new queues (#5179)

* add v3.3.0 operator commits (#5143)

* [COST-5124] Improve Trino migration management command (#5163)

* Add exponential backoff and logging to retries
* Change log level to reflect severity
* Explicit SQL alias for clarity
* Catch and log exception instead of exiting
* Add return type hints
* Return if unsuccessful
  No point in verifying if the SQL did not run correctly
* Fine tune exponential backoff
* Create action class for adding/verifying columns were added
* Assign default list using default_fatory
  Instead of doing it in the post_init, which get’s a little weird.
* Add drop column action
* Quote items in logs for better legibility
* Consolidate action classes
  We lose some of the action-specific logging messages, but there is less
  code overall. I’m not sure how this scale to the action related to dropping.
* Change local variable name
  No need to add a prefix to differentiate it from the parameter name.
* Use a set to prevent running on the same schema multiple times

Co-authored-by: Cody Myers <[email protected]>

* Filter accounts by matching criteria during subs processing to prevent unnecessary SQL from running (#5184)

* Update tasks.py (#5185)

* clean up grafana dashboard (#5183)

* Skip OCPCloud tag SQL if key is present in cache but value is None (#5186)

* [COST-5196] - Send OCP tasks to correct queues (#5187)

* [COST-5196] - Send OCP tasks to correct queues

* [COST-5176] correctly pass context dictionary within log_json function call (#5182)

* [COST-5176] correctly pass context dictionary within log_json function call

* add unittests for exceptions in generate_report

* batch delete S3 files (#5180)

* Bump urllib3 from 1.26.18 to 1.26.19 in the pip group across 1 directory (#5172)

Bumps the pip group with 1 update in the / directory: [urllib3](https://github.com/urllib3/urllib3).


Updates `urllib3` from 1.26.18 to 1.26.19
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.19/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.18...1.26.19)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: indirect
  dependency-group: pip
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add flower as a dev dependency (#5189)

* Add docs

* [COST-4844] Serializer update for ordering by storageclass (#5174)

* Switch to using podman in build_deploy (#5193)

The VM used in CI is now RHEL 8

* skip polling providers still processing (#5181)

* skip polling providers that are still processing

* [COST-5214] Move TARGETARCH declaration to the top of the Dockerfile (#5195)

There is a bug in podman where this is only used correctly for the
multi-stage build if it is defined as the first line.

Update Jenkinsfile to use RHEL 8

Unfortunately this breaks the image build for Docker. I'll fix that in a followup PR.

* [COST-5213] - fix S3 prepare (#5194)

* Switch default parquet flag to prevent iterating on all files in each worker when there is nothing to delete

* [COST-5214] pass build-arg to docker build command (#5196)

* [COST-5216] Delete filtering optimization (#5197)

* Revert "[COST-5216] Delete filtering optimization (#5197)" (#5200)

This reverts commit 97ba98e.

* [COST-5226] - Skip S3 delete (daily flow) if we have marked deletion complete. (#5198)

* dont attempt more S3 deletes if we have marked deletion complete

* [COST-5076] upgrade to python 3.11 (#4444)

* upgrade to python 3.11

* pipfile update

* add gcc-c++ compiler

Co-authored-by: Sam Doran <[email protected]>

* update test

* replace gcc with gcc-c++

---------

Co-authored-by: Sam Doran <[email protected]>

* [COST-5228] log outside for loop (#5202)

* [COST-5228] log outside for loop

* additional log clean up

* add context to logs in _remove_expired_data func

* log s3 batch deletes (#5204)

* log s3 batch deletes

* [COST-5219] Correctly report VM usage for metering when billing record is split (#5201)

* [COST-5219] Handle Azure instance record being split

* [COST-4745] OCPGCP Network data processing SQL (#5058)

* [COST-4745] OCPGCP Network data processing SQL

---------

Co-authored-by: Sam Doran <[email protected]>

* [COST-5198] - split read traffic to read replica db using nginx proxy (#5188)

* update nginx with HTTP method routing
* switch koku-api to koku-api-writes
* duplicate koku-api to koku-api-reads add a optional mounted secret for the read replica
* update clowder configurator to read from read replica secret if mounted and enabled via ENV var

* remove unused methods (#5208)

* Bump certifi in the pip group across 1 directory (#5207)

* chore(image): update and rebuild image (#5203)

Co-authored-by: Update-a-Bot <[email protected]>

* Handle case when resource ID cannot be obtained (#5209)

* Catch exception case.

* [COST-5148] filter out empty resource ids and SavingsPlanCoveredUsage entries (#5206)

* [COST-5148] update insert sql
filter out empty resource ids
offset savings from SavingsPlanCoveredUsage

* closing CASE statement

* clean up comment

* remove case stmts in favor of filtering out SavingsPlanCoveredUsage

* clean up

* Unpause the csi volume handle sql (#5175)

* update linting

* feat: cleaning up old changes. making first changes to create the API pieces.

* feat: cleaning up old changes. making first changes to create the API pieces.

* feat: insert new filters.

* feat: insert new filters and order by params.

* feat: customizing provider map and serializer.

* fix: changing TIME_CHOICES options.

* feat: first unit tests.

* feat: wip.

* feat: fixing provider map.

* feat: fixing provider map.

* update ec2 annotations to get all required fields

* use AWSEC2ComputeQueryParamSerializer

* feat: creating orderby and groupby serializers for ec2

* feat: unit tests for filters

* feat: removing group_by - not needed on ec2

* feat: fixing orderby serializer and starting units tests

* fix: typo

* feat: changing usage_hours to usage_amount

* feat: fixing unit test.

* wip: blocking some filters and unit test.

* feat: unit tests for group by filter

* flake8 fix

* feat: inserting more filters on validate function.

* feat: updating validate method to use similar logic and add filters.

* fix: changing unit tests for some filters.

* feat: testing filter combinations and flake8 checks.

* fix: test

* feat: serializer Unit tests and view Unit test fix

* flake8 fix

* fix: new approach to satisfy CodeCov

* fix:: getting rid of validate custom method.

* fix: commenting tags.

* fix: validarte functions, tests.

* handle filter params for specific report type

* transform tags to desired ui format

* default to monthly resolution on the EC2 endpoint

* add special pagination for EC2

* use default report type time period settings if exists

* fix typo

* fix: fixing parameters validations.

* update linting

* squash commits

* clean up query params and time period settings

* do not use filter keyword

* more code clean up
update unit tests

* address feedback
- move report_specific filters to main filter map
- use report_type instead of kwargs in get_paginator
- do not use deepcopy - just overwrite query_data
- resolution and time_scope_units are always monthly and month respectively
- overide start and end date params in base ParamSerializer
- overide limit and offset in base FilterSerializer

* more unit tests

* update openapi spec

* clean up and add unit tests

* move changes to openapi spec to a separate pr

* use serializer choice field and not customer validate method

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: David N <[email protected]>
Co-authored-by: Luke Couzens <[email protected]>
Co-authored-by: Cody Myers <[email protected]>
Co-authored-by: Corey Goodfred <[email protected]>
Co-authored-by: Sam Doran <[email protected]>
Co-authored-by: Michael Skarbek <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Hambridge <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Update-a-Bot <[email protected]>